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Abstract 


This paper highlights the development of a model that is focused on the safety issue of increasing complexity and 
reliance on automation systems in transport category aircraft. Recent statistics show an increase in mishaps related 
to manual handling and automation errors due to pilot complacency and over-reliance on automation, loss of 
situational awareness, automation system failures and/or pilot deficiencies. Consequently, the aircraft can enter a 
state outside the flight envelope and/or air traffic safety margins which potentially can lead to loss-of-control (LOC), 
controlled-flight-into-terrain (CFIT), or runway excursion/confusion accidents, etc. The goal of this modeling effort 
is to provide NASA’s Aviation Safety Program (AvSP) with a platform capable of assessing the impacts of AvSP 
technologies and products towards reducing the relative risk of automation related accidents and incidents. In order 
to do so, a generic framework, capable of mapping both latent and active causal factors leading to automation errors, 
is developed. Next, the framework is converted into a Bayesian Belief Network model and populated with data 
gathered from Subject Matter Experts (SMEs). With the insertion of technologies and products, the model provides 
individual and collective risk reduction acquired by technologies and methodologies developed within AvSP. 

Introduction 


Usage of automatic systems in airliners has increased fuel efficiency, added extra capabilities, enhanced safety and 
reliability, as well as providing improved passenger comfort since its introduction in the late 80’s. However, original 
automation benefits, including reduced flightcrew workload, human errors or training requirements, were not 
achieved as originally expected. Instead, automation introduced new failure modes, redistributed, and sometimes 
increased workload, brought in new cognitive and attentional demands, and increased training requirements (refs. 1, 
2). Modern airliners have numerous flight modes, providing more flexibility (and inherently more complexity) to 
the flightcrew. However, the price to pay for the increased flexibility is the need for increased mode awareness, as 
well as the need to supervise, understand, and predict automated system behavior (ref. 3). Also, over-reliance on 
automation is linked to manual flight skill degradation and complacency on commercial pilots. As a result, recent 
accidents involving human errors are often caused by the interactions between humans and the automated systems 
(e.g. the breakdown in man-machine coordination), deteriorated manual flying skills, and/or loss of situational 
awareness due to heavy dependence on automated systems (refs. 4, 5). 

This paper describes the development of the increased complexity and reliance on automation baseline model, 
named FLAP for FLightdeck Automation Problems. The FLAP model is part of a series of models that serve the 
NASA Aviation Safety Program’s (AvSP) portfolio assessment by providing simulation capability for complex 
aviation accidents at the system-level. These models 1 provide quantitative analysis capability, enabling the AvSP to 
assess the portfolio impact on the reduction of aviation system risk in the current day operations 2 (ref. 7). The focus 
of the FLAP model is on the effects of increased complexity and reliance on automation systems in transport 
category aircraft accidents and incidents. Consequently, the model aims to simulate contributors associated with 
man-machine interface breakdown, flightcrew manual flight skill degradation, automation interface, overconfidence/ 
complacency and simulator training as well as automated aircraft systems failure and design. 

Data and Literature Review 


In accidents involving late model airliners, the essence of pilot error accidents is no longer related to “stick and 
rudder” or manual flying skills, rather, it is the efficiency of (system) monitoring of highly automated aircraft (refs. 
2, 3, 8). Consequently, pilot error usually results in misalignment of automation system/modes, pilots’ perceptions 


1 The first model consists of aviation accidents caused by in-flight loss of control, captured in a model named LOCAF (ref. 6). 

“ Future versions of these models will provide operations in NextGen environment. 


and actions, and aircraft state. Currently, common taxonomies and definitions for accident and incident * * 3 reporting 
systems such as the CAST/ICAO Common Taxonomy Team’s (CICTT) Aviation Occurrence Categories or the 
accident types in National Transportation Safety Board (NTSB) categories don’t have a dedicated group concerned 
with autonomy/automation related issues 4 5 . Lack of a dedicated category prevents a comprehensive search of the 
dataset of such accidents, which, in turn, prohibits a statistical analysis capability on automation related accidents 
with respect to all accidents within a certain database/time frame. Due to these shortcomings, accident and incident 
data found in literature are solely used to identify key issues and provide information during framework 
development, and served as guidelines/examples in SME meetings. 

A literature review on issues associated with increased automation and its effects on flightcrew was conducted. The 
available literature mostly consisted of anecdotal work; describing main problem areas associated with increased 
automation usage. One of the most cited works; the Federal Aviation Administration (FAA) Human Factors Team 
Report (ref. 10) addresses flightcrew/flight deck automation interfaces in commercial aircraft, and provides 
comprehensive information on the issues and recommendations 3 . Studies conducted by Billings (ref. 2), Sarter et al. 
(ref. 3), and Orlady et al. (ref. 11) also shed light on earlier issues encountered in automated systems as well 
evolution of aircraft automation, which assisted in identifying primary issues simulated in this model. Also, studies 
investigating incidents (refs. 4, 10, 12) pilot surveys that collect information on pilots’ attitudes about flight deck 
automation (refs. 13, 14) were used to construct the framework structure. 

Modeling Steps Overview 

Employing the literature and accident/incident data review, a comprehensive list of causal factors contributing to 
automation problems was acquired and categorized based on responsible parties (e.g., flightcrew, regulatory body, 
etc.). These causal factor categories were then organized within a hierarchical manner; similar to Reason’s Swiss 
cheese model (ref. 15) used in the Human Factors Analysis and Classification System (HFACS) (ref. 16) and the 
previous modeling effort, LOCAF (ref. 6). The causal factors (or nodes) and the connecting links within the 
framework are supported and documented by both past studies and accidents/incidents alike. The resultant 
framework is a generalized representation of automation related accidents/incidents, capable of showing multi- 
dependencies among various automation stakeholders. The next step involved the conversion of the framework into 
a quantitative model using a Bayesian approach via Hugin Software v.7.8 6 . The draft model was reviewed by 
Subject Matter Experts (SMEs) in order to obtain feedback, validation, and probabilistic data. Following subsequent 
calibration, the resultant model, called “baseline model”, is capable of providing preliminary causal factors and their 
probability values leading to an automation accident and/or incident. Finally, the model will be reviewed by internal 
and external panels before the AvSP products are inserted for portfolio assessment purposes. The model will then 
provide the effect of portfolio elements (also called products) in reducing automation related events in today’s 
aircraft operations. 


FLAP Framework 


The FLAP framework is structured to contain both latent and active factors. Active failure levels are the 
‘pointy/sharp end’ of the spear where the event takes place on the front line operator level, and are often directly 
linked to the accident or incident. Deficiencies or failures on each level are viewed as ‘holes’. Undesirable events 
are caused by overlapping of these failures/breakdowns (holes) at latent and active layers (ref. 16). The FLAP 
framework containing three latent and two active levels and their interactions are given in Figure 1 . 


NTSB defines accident an occurrence associated with the operation of an aircraft, which takes place between the time any 

person boards the aircraft with the intention of flight and all such persons have disembarked, and in which any person suffers 

death or serious injury, or in which the aircraft receives substantial damage and, and an incident is an occurrence other than 
accident, associated with the operation of an aircraft, which affects or could affect the safety of operations (ref. 9). 

4 ASRS incident database includes “human-machine interface” under human factors tab as a direct automation related factor 
since June 2009. 

5 The Flightdeck Automation Working Group (FDAWG) (ref. 4) recently released an updated version of the 1996 report and 
findings which were compared against the FLAP model. 

6 http://www.hugin.com/productsservices/products/commercial/explorer 
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Figure 1 - FLAP Framework Overview 


The first latent level (LI) includes the major stakeholders within commercial airline operations, i.e. regulatory body 
(FAA, Directorate General for Civil Aviation, etc.), aircraft manufacturers, and operators. Second level latent (L2) 
factors include issues related to high-level underlying factors including automation characteristics such as design, 
interface, and reliability, as well as airline policy/procedures and training practices. The third level consists of latent 
flightcrew (FC) related factors such as complacency/trust, understanding & system knowledge, flight skills 
degradation, and experience/background. There are two active causal factor levels in the framework. The first level 
active causal factors, (also considered as ‘triggers’ or precursors), are divided into two sub-sections. Active A1 A and 
Active A1B. These active failures take place during flight and stem from either automation system anomalies (A1 A) 
or the underperforming flightcrew (A1B) due to several reasons. The second active level (A2) represents automation 
mishap including flightcrew errors (FC awareness/monitoring, decision deficiency, and automation surprise) along 
with flight anomaly and recovery. The contents of the active as well as latent levels are provided within the node 
descriptions in the next section. 


FLAP Model Overview and Node Descriptions 

The framework presented in the section above was converted into a Bayesian Belief model using Hugin Software. In 
order to facilitate its representation and discussion, the model was divided into three sections using Hugin’ s object- 
oriented feature 7 which allows encapsulation of certain parts of the model. The FLAP model consists of the top- 
level in Figure 2 and encapsulated Automation and Flightcrew Conditions sections (called subnets) given in Figure 3 
and Figure 4, respectively. 


7 Note that Hugin Software allows encapsulation only if subnets do not receive any inputs from top level nodes, a requirement of 
directed acyclic graph structure of Object-Oriented Bayesian Networks (OOBNs) to prevent directed cycles. In order to 
overcome this constraint and for the sake of discussion, the common causal factors nodes are duplicated in all three model 
sections/subnets. Actual model calculations are performed on a “flat” model, without subnets. 





Top-level FLAP Model : The top-level model is used to integrate the subnets and convey the causal factors to the 
event of interest, i.e. probability of an automation related accident and incident. Similar to the structure shown in 
Figure 1, the top-level model, shown in Figure 2, includes several layers of latent and active factors and their 
interactions, described in detail in the sections below. 



Figure 2 - Top-Level FLAP Model 


The bottom three nodes include the LI level organizational latent factors such as the Regulatory Body, 
Manufacturer, and Operators/Airlines. The LI level nodes and their respective links are duplicated in Automation 
and Flightcrew Conditions subnets, but are only described in this section. The Regulatory Body node represents 




deficiencies within the regulatory process in both aircraft certification and flight standards of commercial transport 
operations (ref. 4). The Regulatory Body node is linked to five other nodes: Manufacturer, Training, and 
Operators/Airlines (Figure 2), Automation Design, Automation Interface (links shown in Figure 3). The 
Manufacturer node represents large aircraft manufacturers as well as automation system/avionic equipment 
manufacturing companies. The node covers deficiencies in automation design philosophy and approach (ref 11), 
level of automation including over-automation (ref. 13), economic benefit (ref 17), and standardization and cultural 
diversity (refs. 10, 18) that could eventually lead to human factors issues. The Manufacturer node influences 
characteristics of Automation Design, Automation Interface (links shown in Figure 3), and also affects Training and 
Policy/Procedures nodes (ref. 10) (Figure 2). The Operators/Airlines Node delineates the organizational aspects of 
corporate airlines as causal or contributing factors in automation accidents/incidents.. The organizational 
deficiencies can trickle down and materialize as Training (refs. 10, 14, 17) or Policy/Procedure (ref. 10) issues 
(Figure 2). Also, lack of adequate supervision and management guidance can result in Adverse Physiological or 
Mental States in flightcrews, links shown in Figure 4. Finally, link between the Operator/Airlines and Manufacturer 
nodes represent unrealistic airline expectations/requirements imposed on the manufacturers, driven by economic 
motivation and operational efficiency. 

The second latent layer includes underlying factors affecting both automation systems and airline operations. These 
factors are Policy/Procedures and Training in the Top-Level Flap model in Figure 2, and Automation Design, 
Automation Interface, and Automation Reliability (given in the automation subnet. Figure 3). The Training node 
includes deficiencies associated with inadequate training, generally associated with limited resources and the 
common practice of “on-the-job” training of the remainder automation functions that were left out during initial 
training (refs. 3, 10, 11). This node is connected to several causal factors in the model. Different components of 
training are represented via the node it is linked to, for instance Training to Flight Skills Degradation designates the 
basic flight (stick & rudder) component of training. Similarly, Training is connected to 
Awareness/Monitoring/Scanning/Attentional (Awareness/Monitoring for short) and Understanding/ System 
Knowledge nodes as shown in Figure 2 and Crew Resource Management (CRM) node in Figure 4, representing 
respective training constituents. The Policy/Procedures node covers deficiencies associated with inappropriate 
Rightcrew guidance caused by issues stemming from Manufacturer and Operators/Airlines. For instance, some 
examples of inadequately determined procedures are: operator procedures inconsistent with manufacturer 
recommendations and design philosophy, incorrect modification of procedures for economic or fuel saving reasons, 
or inappropriate level of proceduralization (refs. 4, 10). The Policy/Procedures node is connected to the Flight Skills 
Degradation and Trust/Reliance nodes to represent cases where operator guidance on automation usage level solely 
promotes automatic flight aircraft handling. Similarly, inappropriate guidance on aircraft control could interfere with 
pilot understanding and system knowledge and affect training procedures. 

The third level of latent causal factors consists of Flightcrew Experience/Background, Trust/Reliance, Flight Skill 
Degradation, Understanding/ System Knowledge nodes. The FC Experience/Background node distinguishes the 
varying flightcrew personal factors including experience level (seasoned vs. young pilots), training background 
(military vs. civil aviation), and operational environment. The Flightcrew Experience/Background node is linked to 
Flight Skills Degradation, Understanding/ System Knowledge, and flightcrew Trust/Reliance nodes (refs. 3, 17). The 
Trust/Reliance node includes flightcrew complacency and inappropriate confidence level assigned to autoflight 
systems. The Trust/Reliance node is a parent causal node for three nodes, namely, Awareness/Monitoring (ref. 3), 
Flight Skills Degradation (ref. 19), and Decision Deficiency (ref. 17). The Flight Skills Degradation node contains 
erosion of manual flight skills (e.g. basic stick-and-rudder capabilities, flight control errors, instrument scan, etc.) 
due to continuous operation of autoflight systems and lack of practice (refs. 5, 1 1). The Flight Skills Degradation 
node is connected to both the Flight Anomaly and Final Recovery nodes to represent the cases where degraded skills 
can cause flight anomalies or unsuccessful flight recovery. An additional link from Dynamic FC Conditions subnet 
was added to provide crew’s current status including physiological, mental factors and distractions. The 
Understanding/System Knowledge node refers to issues related to flightcrew knowledge of aircraft systems or 
presence of gaps and misconceptions in their mental model of the system (e.g. autoflight modes, flight management 
computer, system couplings). This node is connected to the Decision Deficiency node since issues with pilot 
Understanding/System Knowledge is one of the primary sources of errors captured in the Decision Deficiency node. 
Pilot understanding is also linked to Automation Surprise and flightcrew System Awareness/Monitoring nodes, (ref. 
10 ). 


The A2 active failure level includes nodes associated with flightcrew System Awareness/Monitoring, Decision 
Deficiency, Automation surprise which lead to Flight Anomaly, and Final Recovery. The Awareness/Monitoring 
node includes insufficient system awareness, defined as the inability of a supervisor to track and anticipate the 
behavior of a) the automation system variables and controls, b) aircraft state and flight parameters (position, speed, 
flight path, energy state), and c) operating environment (terrain, air traffic clearances, and traffic) (refs. 1, 3, 10). 
This node is linked to two nodes; Decision Deficiency (refs. 10, 17) and Automation Surprises (ref. 3). The Decision 
Deficiency node includes all cognitive errors made by the flightcrew such as mode selection/confusion error, flight 
management system (FMS) programming, checklist use/procedures errors, misdiagnosis of faults, etc. The Decision 
Deficiency node influences three nodes; Automation Surprise (due to improper programming or mode errors), Flight 
Anomaly, and Final Recovery (representing deficiencies in recognizing the anomaly and selecting proper mitigation 
strategy). The Automation Surprise node defines cases where the operator is surprised by the automation systems, 
unable to comprehend its current behavior or estimate future occurrences (ref. 1). The presence of automation 
surprises is one of the prominent causal factors for the Flight Anomaly node where the flightcrew recognizes that the 
aircraft is outside its flight envelope or restrictions via cues from aircraft systems or air traffic control (ATC) 
interventions. The Flight Anomaly node designates any departure from the intended flight plan or safe flight 
envelope that qualifies as an incident. These anomalies include aerospace deviation in altitude, speed, position, 
aircraft performance parameters and energy management deficiency as well as aircraft entering a flight state without 
being properly configured. Depending on the anomaly, the aircraft could potentially experience stall, loss-of- 
control, over-speed, loss of separation (and consequent near mid-air collision, mid-air collision), controlled flight 
into terrain, or other accidents/incidents. The Final Recovery node refers to the ability of the flightcrew to recover 
from an abnormal flight condition defined in the Flight Anomaly node. Given that the model simulates an accident/ 
incident environment, the Final Recovery node plays a decisive role on whether the incident turns into an accident. 

Automation Subnet : As previously discussed, issues stemmed from automation systems are compiled under the 
Automation Subnet given in Figure 3. Similar to the top-level model, the automation subnet also follows the 
active/latent causation structure, leading to active errors/issues. LI level latent organizational factors including 
Regulatory Body, Manufacturers, and Operator/Airlines node descriptions are given above. The L2 level latent 
underlying factors include the Automation Design, Automation Reliability and Automation Interface nodes. The 
Automation Design node encompasses issues within the automation system design process including system, 
hardware, and software designs from preliminary phase to flight hardware including assumptions, requirements, 
testing/debugging, implementation, verification and validation, quality assurance, etc. (ref. 20). The Automation 
Design node is connected to all the downstream nodes in the automation subnet since improper planning and 
execution of automation requirements can result in failures and unexpected behavior throughout the system. 

The active layer within the automation subnet includes Operating Environment, Hardware/Software Failure, and 
five automation function nodes. Operating Environment node provides external causes that potentially affect the 
operation of hardware/software (HW/SW) either by disrupting sensor outputs (e.g. cold weather, ice, volcanic ash) 
or by damaging aircraft systems directly (e.g. lightning, impact damage). Similarly, the HW/SW Failure node 
includes all glitches and malfunctions of the systems that were not anticipated by the designers, including 
malfunctions of antennas, sensors, or other measuring equipment that provide information to the automation systems 
downstream nodes. The Performance Systems node includes issues associated with the performance function of the 
FMS, specifically, weight and balance, fuel weight, engine thrust limits, take off reference data, maximum/optimum 
altitude calculations, or carrying out the projected altitude or speed targets (ref. 21). The Warning & Monitoring 
Systems include automated warning systems such as aircraft configuration, monitoring of aircraft systems and 
presence of environmental threats (ref. 11). This node covers the failure of warning systems due to both faulty 
design and/or HW/SW failure stemming from the design and implementation of these systems. The issues related to 
ergonomic aspects of the warning systems (e.g. selection and characteristics of visual, auditory or tactile alerting 
systems) are included in the Automation Interface node. The Navigation Systems node covers the components and 
systems used in navigation including the global positioning system, Very High Frequency omnidirectional radio 
range, and other precision approach system components (ref. 21). The Flight Control Systems node encompasses all 
the systems involved in automatic flight within the FMS and implementation of inputs via flight control surfaces. 
The node also includes authority and autonomy related issues and assumptions such as envelope protection and 
stall/bank limiters (ref. 21). Finally, the Communication Systems node includes data link and surveillance systems. 
Systems like the Aircraft Communications Addressing and Reporting System, telemetry, communication radios, 
satellite links, telemetry, and ADS-B/C are included in this node (ref. 21). 



The Automation Subnet provides three output nodes 8 that are transferred to the top-level FLAP model. The first 
active output node. Automation Issue, provides the probability of an automation system exhibiting malfunction or 
failure, stemming from any of the five functional systems described above. The Automation Reliability node is L2 
level output of the subnet, stemming from the Automation Design node. Automation reliability primarily affects the 
flightcrew’s perception where highly reliable automation systems inherently increase reliance on automation 
(connected to Trust/Reliance node at the top-level). The final output node of this subnet is the Automation Interface. 
The interface node is identified as one of the most prominent causes of man-machine breakdown due to its effects on 
flightcrew situational awareness, pilot saturation and/or confusion (refs. 2-4, 10, 22). This node contains the human- 
factor related aspects of the cockpit design such as inappropriate determination of the characteristics of visual, 
auditory, and tactile alerting systems as well as alert categorization issues (ref. 22). This node also includes the non- 
intuitive flightcrew interface including inadequate feedback and standardization. The Automation Interface node is 
linked to three causal factors; Trust/Reliance, Awareness/Monitoring (refs. 3, 10), and Understanding/System 
Knowledge (ref. 4) as shown in Figure 2. 

Flightcrew Conditions Subnet : The flightcrew conditions subnet provides dynamic flightcrew conditions to the top- 
level FLAP model, in an approach similar to Reason’s model of accident causation. As in the automation subnet, 
the flightcrew conditions subnet includes active and latent layers, providing one output node to the top-level FLAP 
model. 


s The output nodes have gray-shaded borders, as shown in Figure 3. 



Similarly to the Automation Subnet, the LI and two L2 latent layer nodes ( Regulatory Body, Operational/Airlines, 
Manufacturer, Policy/Procedures, and Training ) are duplicated in this subnet. Adverse Physiological and Mental 
States and FC Preconditions nodes in this subnet belong to A1 level pre -flight causal factors and help determine the 
probability of flightcrew readiness before the flight takes place. The Flightcrew Preconditions node is an 
aggregation node that considers flightcrew adverse physiological and mental states as well as CRM deficiencies and 
determines how fit the flightcrew is for the upcoming flight. This node is linked to the Dynamic Flightcrew 
Conditions node, providing a baseline which is then updated by considering the presence of distractions throughout 
the flight. The Adverse Physiological States node include physical fatigue/lack of sleep, medical illness, impaired 
physiological state, physiological incapacitation, self-medication, violation of crew rest requirement (ref. 6). The 
node is connected to the FC Preconditions node which then connects to Awareness/ monitoring node which is 
greatly affected by the presence of physiological issues. Adverse Mental States node include complacency, 
distraction, get-home-itis, misplaced motivation, mental tiredness, distraction, confusion, depression, and/or 
alcoholism (ref. 6). The presence of adverse mental states can be the cause of decision deficiencies (via the Dynamic 
FC Conditions node) which include omission errors. Crew Resource Management (CRM) node includes deficiencies 
like communication skills and coordination that take place among the flightcrew as well as between other entities 
before, during, and after the flight (ref. 6). CRM deficiencies surface as cross-verification errors and crew 
coordination problems including workload management. The node is linked to both FC Preconditions and Dynamic 
FC Conditions, indicating issues associated with the pre-flight briefing as well as in-flight communication and 
coordination, respectively. 



A1 level in-flight causal factors including A/C System Distraction and Distraction provide the model with the 
updated flightcrew conditions that are present during the flight. Along with the input from the FC Preconditions 
node and several sources of external distraction, the ability of the crew to perform flying duties is aggregated in the 
Dynamic FC Conditions node and outputted to the top-level model. Possible distractions the flightcrew may 
encounter during a flight are divided into two categories; general distractions and aircraft system related distractions. 
The Aircraft (A/C) Systems Related Distractions node is an aggregation node which provides the probability of 
flightcrew to get distracted by the presence of a) Automation Issues, and b) SCF. Distractions stemmed from 
troubleshooting autoflight system anomalies/behavior as well as reprogramming the FMS are captured in this node. 
The SCF related node provides all other failures within the aircraft systems that are not associated with autoflight 
systems. Another aggregation node. Distractions, takes into consideration all other major sources of mental 
disturbance that potentially result in fixation or absorption. The causal factors considered are fourfold; Traffic, ATC, 
On-board Personnel, and Weather. The presence of traffic and frequent changes in the flight trajectory were 
identified as sources of distraction in several incidents (ref. 1, 12). Besides ATC and traffic, numerous cases where 
cabin crew interference with cockpit or presence of check pilot during flight were also identified as causes for the 
flightcrew overlooking the autoflight systems (ref. 12). Lastly, the presence of adverse weather (icing, fog/visibility 
issues, thunderstorms, rain, wind, and low ceiling) causes additional mental work to the flightcrew (ref. 12). 
Presence of distractions inherently increases probability of flightcrew experiencing fixation or saturation since they 
require identification and mitigation. 

As previously mentioned, the Dynamic Flightcrew Conditions node is the sole output of this subnet and it is linked 
to key causal factors including Decision Deficiency, Flight Skills Degradation, and Awareness/ Monitoring. This 
node takes into consideration the presence of distractions along with crew related issues in order to calculate the 
probability of a crew member suffering fixation or absorption. Fixation is described by “being locked into one task 
or one view of a situation even as evidence accumulates that [...] the particular view is incorrect” (ref. 10, p.59). On 
the other hand, absorption is a state of mind where the pilot is focused on one single task such as FMS programming 
or flight management computer troubleshooting while discarding others (also referred to as task shedding). 

FLAP Model Output Nodes : The output values of the FLAP model (shown in Figure 2) are determined as the 
incident and accident probabilities associated with automation related issues. The model is designed to identify the 
prominence of automation related issues among all foreseeable accidents and incidents. For that reason, by 
definition, the Flight Anomaly node provides the probability of an in-flight upset resulting from the combination of 
the upstream automation-related nodes. The Automation Related Event Probability node reflects all automation 
related events including both incidents and accidents. This node is comprised of automation related events (A%) and 
non-automation related events (B%). A% and B% sum up to 100%, representing all considered accidents and 
incidents. Automation related Flight Anomaly node probability combined with that of the Decision Deficiency and 
Flight Skills Degradation nodes are the inputs to the Final Recovery node, which determines the probability of an 
incident evolving into an accident. The assumption is such that recovered incidents remain as incidents (e.g., 
correction of over-speed or stall situation) where, unsuccessful recovery efforts may result into injury to crew and 
passengers and/or damage to the aircraft, hence an accident. Consequently, the second output node. Automation 
Related Incident/Accident Probability, provides accident and incident probabilities (e.g., X% accidents and Y% 
incidents, totaling up to 100% of flight anomalies or A%). Given the risk is defined as the product of likelihood of 
event and its severity, the model provides risks associated with heavy automation in today’s aircraft operations. This 
is achieved by providing likelihood (probabilities) of such events occurring (via Automation Related Event 
Probability node) and their severity (via Automation Related Incident/Accident Probability node which distinguishes 
between accidents and incidents). 


AvSP Product Insertion Process 


NASA’s AvSP is responsible for developing methodologies and technologies (referred to as products) to improve air 
transportation safety within the NextGen environment. The AvSP is comprised of three projects, namely. Vehicle 
Systems Safety Technologies project, System-Wide Safety and Assurance Technologies project, and Atmospheric 
Environment Safety Technologies project (ref. 23). As previously discussed, the goal of the modeling effort is to 
gauge the impact of the products developed within these projects on current and future aviation risks. In order to do 
so, the products will be inserted into the model by the SMEs and then verified by involved stakeholders to ensure 
proper placement. The next step involves SME re-evaluation of affected nodes’ probability by considering the effect 
of the product. The rectangular nodes shown in Figure 5 represent notional products and the arrows designate the 


affected nodes. Owing to the Bayesian Belief structure, the benefits of the products are propagated downstream from 
the affected node. For instance, a new technology that improves pilot situational awareness is applied at the 
Awareness/Monitoring node and can prevent a Flight Anomaly. 



Data Collection 


The probability data elicitation process consists of a series of SME sessions. The model structure and probability 
values as well as the impacts of AvSP portfolio elements are acquired from the same set of SMEs, called operational 
SMEs. For the FLAP model, the operational SMEs consist of two commercial pilots; experienced in Part 121 and 
135 operations and airline management/training as well as one human factors expert specialized in flight deck 
automation. The additional SME panels held in between operational SME meetings are used to ensure the model 
assumptions and structure are sound for the given study purpose. A typical elicitation process for the modeling effort 
is highlighted in Luxhpj (ref. 24). At the time of writing, the baseline model was established following the first 
operational SME meeting. The goal of this four-day meeting was to review the preliminary model developed by the 
team, make revisions, and populate probabilities for the causal factors 9 . 

Boundary Conditions : As previously stated, the model is intended to solely capture the probability of automation 
related flight anomalies that could result in incidents and accidents. The two top-level output nodes provide accident 
and incident probabilities of automation related anomalies with respect to all conceivable accidents and incidents. 
For that reason, the elicitation process strictly considers probabilities within the accident/incident perspective instead 
of all aviation operations when querying causal factor probabilities in the model. This assumption renders the 
probabilities more tangible; e.g. probability of situational awareness deficiency with respect to all yearly U.S. -based 
flights (over 8 million departures in 2013 10 alone) versus all accidents and incidents within the last 10 years (over 
1200 accidents * 11 in Part 121 and 135 and 35000 incidents in ASRS database 12 ). The model considers today’s 
aircraft operating within the United States, under FAR Part 121 & FAR 135 with considerable but varying degrees 
of automation usage, such as Boeing 737, -47, -57, -67, -77, -87 families and Airbus 300, -10, -20, -30, -40, -50 as 
well as regional jets like Embraer and Bombardier CRJs. The timeframe for this study was a 10 year period 
including aircraft commissioned in year 2003 until 2013. 


9 Since the model results were uncalibrated, the preliminary numerical values were not included in the context of this paper. 
1,1 http://www.transtats.bts.gov/ 

1 1 http://www.ntsb.gov/aviationquery/ 

12 http://akama.arc.nasa.gov/ASRSDBOnline/QueryWizard_Filter.aspx 







Assumptions and Limitations : There were several assumptions made throughout the modeling effort due to limited 
resources. As in the previous modeling effort and some other BBN approaches, the FLAP model embraces SME 
opinion for the sole source for data generation primarily due to lack of statistically meaningful data. Although 
several accidents, incidents and studies were used to develop the model structure, the required probability values in 
the model were acquired from the SMEs. Additionally, in order to keep the model size manageable and still achieve 
a generalized automation problem model, nodes like Awareness/Monitoring and Decision Deficiency contain several 
assumptions and error types, lowering the model resolution. However, since the model is primarily used to evaluate 
future NASA technologies’ impact as part of a system-level comprehensive portfolio analysis study, the model 
resolution and fidelity satisfy the analysis purpose. Consequently, the model output is not intended to assist 
flightdeck design or policy decision making processes. Also, due to the variance in application within the industry, 
the automation system was developed based on major functions instead of actual system components. 

Conclusions & Next Steps 

This paper highlights the development process of a high-level automation related accident/incident model aimed to 
serve as a platform for AvSP portfolio assessment. In order to do so, past automation studies and accidents/incidents 
were reviewed and key issues were identified. Similarly to LOCAF modeling effort, these key issues are then 
represented in a hierarchical manner and their interdependencies were mapped within the FLAP framework. The 
network was then modeled using the Hugin Software. In order to populate the model, SME opinions were employed 
due to lack of a comprehensive historical dataset. 

The next steps in the current modeling effort consist of an external review to check the baseline model soundness 
and validity, followed by the second operational SME session allocated for model calibration, review of 
concerns/comments provided by the external panel, and insertion of AvSP portfolio products into the model. 
Following a set of internal review meetings, the third operational SME meeting is planned to revisit the model and 
provide updated probabilities for inserted AvSP products and their impacts. The analysis of the data stemming from 
the model provides insight on a) increased automation dependence on today’s aircraft and its implications, and b) 
impact of NASA products in mitigating such issues. Finally, the FLAP model will be integrated into the past and 
future modeling efforts owing to the OOBN modeling techniques and will be further used in portfolio prioritization 
efforts. 
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